Detecting Anomalous Application Behaviors Using a System Call Clustering Method over Critical Resources

Malware attacks which focus on exploiting an application to launch the payload have become major security threat. We present the methodology and algorithm which is able to detect anomaly in application behavior and prevent such type of attacks. Our approach is to represent the normal behavior of an application, detect deviations from this normal behavior and prevent them. We represent normal behavior using system calls made over critical resources by clustering of these system calls and then monitor the behavior of applications for any deviations from the normal behavior, by means of an enforcement algorithm. Any mismatch from the normal behavior indicates an anomaly. We provide a description of our approach. We have implemented and tested the proposed approach and the results are encouraging. As compared to previous research in this direction, we implement on Windows OS instead of Linux OS and use minifilter and registry callback techniques instead of raw system call interception which is prohibited in latest operating system versions.